{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Xception_implementation.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyNq9qTKVN05k/mMCHpbG6D2",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Machine-Learning-Tokyo/CNN-Architectures/blob/master/Implementations/Xception/Xception_implementation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aBMbYREY0Jt8",
        "colab_type": "text"
      },
      "source": [
        "# Implementation of Xception\n",
        "\n",
        "We will use the [tensorflow.keras Functional API](https://www.tensorflow.org/guide/keras/functional) to build Xception from the original paper: “[Xception: Deep Learning with Depthwise Separable Convolutions](https://arxiv.org/abs/1610.02357)” by François Chollet.\n",
        "\n",
        "[Video tutorial](https://www.youtube.com/watch?v=nMBCSroJ7bY&list=PLaPdEEY26UXyE3UchW0C742xh542yh0yI&index=6)\n",
        "\n",
        "---\n",
        "\n",
        "In the paper we can read:\n",
        "\n",
        ">**[i]** “all Convolution and SeparableConvolution layers are followed by batch normalization [7] (not included in the diagram).\"\n",
        ">\n",
        ">**[ii]** \"All SeparableConvolution layers use a depth multiplier of 1 (no depth expansion).\"\n",
        "\n",
        "<br>\n",
        "\n",
        "We will also use the following Diagram **[iii]**:\n",
        "\n",
        "<img src=https://raw.githubusercontent.com/Machine-Learning-Tokyo/DL-workshop-series/master/Part%20I%20-%20Convolution%20Operations/images/Xception/Xception.png? width=\"100%\">\n",
        "\n",
        "<br>\n",
        "\n",
        "as well the following Table **[iv]** to check the total number of parameters:\n",
        "\n",
        "<img src=https://raw.githubusercontent.com/Machine-Learning-Tokyo/DL-workshop-series/master/Part%20I%20-%20Convolution%20Operations/images/Xception/Xception_parameters.png? width=\"40%\">\n",
        "\n",
        "---\n",
        "\n",
        "## Network architecture\n",
        "\n",
        "The model is separated in 3 flows as depicted at **[iii]**:\n",
        "- Entry flow\n",
        "- Middle flow with 8 repetitions of the same block\n",
        "- Exit flow\n",
        "\n",
        "According to **[i]** all Convolution and Separable Convolution layers are followed by batch normalization.\n",
        "\n",
        "---\n",
        "\n",
        "## Workflow\n",
        "We will:\n",
        "1. import the neccesary layers\n",
        "2. write one helper function for the Conv-BatchNorm block and one for the SeparableConv-BatchNorm block according to **[i]**\n",
        "3. write one function for each one of the 3 flows according to **[iii]**\n",
        "4. use these helper functions to build the model.\n",
        "\n",
        "---\n",
        "\n",
        "### 1. Imports\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lhb0t5ZX0CWm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.layers import Input, Conv2D, SeparableConv2D, \\\n",
        "     Add, Dense, BatchNormalization, ReLU, MaxPool2D, GlobalAvgPool2D"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JVBKdFWp0b6J",
        "colab_type": "text"
      },
      "source": [
        "### 2.1. Conv-BatchNorm block\n",
        "The *Conv-BatchNorm block* will:\n",
        "- take as inputs:\n",
        "  - a tensor (**`x`**)\n",
        "  - the number of filters of the *Convolution layer* (**`filters`**)\n",
        "  - the kernel size of the *Convolution layer* (**`kernel_size`**)\n",
        "  - the strides of the *Convolution layer* (**`strides`**)\n",
        "- run:\n",
        "  - apply a *Convolution layer* to **`x`**\n",
        "  - apply a *Batch Normalization* layer to this tensor\n",
        "- return the tensor"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1EmtXA_00fC6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def conv_bn(x, filters, kernel_size, strides=1):\n",
        "    x = Conv2D(filters=filters,\n",
        "               kernel_size=kernel_size,\n",
        "               strides=strides,\n",
        "               padding='same',\n",
        "               use_bias=False)(x)\n",
        "    x = BatchNormalization()(x)\n",
        "    return x"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I38TMjXk0i0t",
        "colab_type": "text"
      },
      "source": [
        "***Note***: We include *use_bias=False* for the final number of parameters to match the ones written at **[iv]**.\n",
        "\n",
        "---\n",
        "\n",
        "### 2.2. SeparableConv-BatchNorm\n",
        "The *SeparableConv-BatchNorm block* has similar structure with the *Conv-BatchNorm* one"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZyOzUA-m0jcF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def sep_bn(x, filters, kernel_size, strides=1):\n",
        "    x = SeparableConv2D(filters=filters,\n",
        "                        kernel_size=kernel_size,\n",
        "                        strides=strides,\n",
        "                        padding='same',\n",
        "                        use_bias=False)(x)\n",
        "    x = BatchNormalization()(x)\n",
        "    return x"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mWLuniSA0nRG",
        "colab_type": "text"
      },
      "source": [
        "### 3.1. Entry flow\n",
        "<img src=https://raw.githubusercontent.com/Machine-Learning-Tokyo/DL-workshop-series/master/Part%20I%20-%20Convolution%20Operations/images/Xception/entry_flow.png? width=\"300\">"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q3Cx3Fc60qZ2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def entry_flow(x):\n",
        "    x = conv_bn(x, filters=32, kernel_size=3, strides=2)\n",
        "    x = ReLU()(x)\n",
        "    x = conv_bn(x, filters=64, kernel_size=3)\n",
        "    tensor = ReLU()(x)\n",
        " \n",
        "    x = sep_bn(tensor, filters=128, kernel_size=3)\n",
        "    x = ReLU()(x)\n",
        "    x = sep_bn(x, filters=128, kernel_size=3)\n",
        "    x = MaxPool2D(pool_size=3, strides=2, padding='same')(x)\n",
        " \n",
        "    tensor = conv_bn(tensor, filters=128, kernel_size=1, strides=2)\n",
        " \n",
        "    x = Add()([tensor, x])\n",
        "    x = ReLU()(x)\n",
        "    x = sep_bn(x, filters=256, kernel_size=3)\n",
        "    x = ReLU()(x)\n",
        "    x = sep_bn(x, filters=256, kernel_size=3)\n",
        "    x = MaxPool2D(pool_size=3, strides=2, padding='same')(x)\n",
        " \n",
        "    tensor = conv_bn(tensor, filters=256, kernel_size=1, strides=2)\n",
        " \n",
        "    x = Add()([tensor, x])\n",
        "    x = ReLU()(x)\n",
        "    x = sep_bn(x, filters=728, kernel_size=3)\n",
        "    x = ReLU()(x)\n",
        "    x = sep_bn(x, filters=728, kernel_size=3)\n",
        "    x = MaxPool2D(pool_size=3, strides=2, padding='same')(x)\n",
        " \n",
        "    tensor = conv_bn(tensor, filters=728, kernel_size=1, strides=2)\n",
        "    x = Add()([tensor, x])\n",
        " \n",
        "    return x"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VdDWkhHO0sAw",
        "colab_type": "text"
      },
      "source": [
        "### 3.2. Middle flow\n",
        "<img src=https://raw.githubusercontent.com/Machine-Learning-Tokyo/DL-workshop-series/master/Part%20I%20-%20Convolution%20Operations/images/Xception/middle_flow.png? width=\"250\">"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h6pBxNcA0y0I",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def middle_flow(tensor):\n",
        "    for _ in range(8):\n",
        "        x = ReLU()(tensor)\n",
        "        x = sep_bn(x, filters=728, kernel_size=3)\n",
        "        x = ReLU()(x)\n",
        "        x = sep_bn(x, filters=728, kernel_size=3)\n",
        "        x = ReLU()(x)\n",
        "        x = sep_bn(x, filters=728, kernel_size=3)\n",
        " \n",
        "        tensor = Add()([tensor, x])\n",
        " \n",
        "    return tensor"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DNd1B5-002C9",
        "colab_type": "text"
      },
      "source": [
        "### 3.3. Exit flow\n",
        "<img src=https://raw.githubusercontent.com/Machine-Learning-Tokyo/DL-workshop-series/master/Part%20I%20-%20Convolution%20Operations/images/Xception/exit_flow.png? width=\"300\">"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VsxFWeQu04bW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def exit_flow(tensor):\n",
        "    x = ReLU()(tensor)\n",
        "    x = sep_bn(x, filters=728, kernel_size=3)\n",
        "    x = ReLU()(x)\n",
        "    x = sep_bn(x, filters=1024, kernel_size=3)\n",
        "    x = MaxPool2D(3, strides=2, padding='same')(x)\n",
        " \n",
        "    tensor = conv_bn(tensor, filters=1024, kernel_size=1, strides=2)\n",
        " \n",
        "    x = Add()([tensor, x])\n",
        "    x = sep_bn(x, filters=1536, kernel_size=3)\n",
        "    x = ReLU()(x)\n",
        "    x = sep_bn(x, filters=2048, kernel_size=3)\n",
        "    x = ReLU()(x)\n",
        "    x = GlobalAvgPool2D()(x)\n",
        "    x = Dense(units=1000, activation='softmax')(x)\n",
        " \n",
        "    return x"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QrfcSKHq07o3",
        "colab_type": "text"
      },
      "source": [
        "### 4. Model code"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9_YCQNXO093K",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "input = Input(shape=[299, 299, 3])\n",
        " \n",
        "x = entry_flow(input)\n",
        "x = middle_flow(x)\n",
        "output = exit_flow(x)\n",
        " \n",
        "from tensorflow.keras import Model \n",
        "model = Model(input, output)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mNGVPgYd4Z7A",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.utils import plot_model\n",
        "plot_model(model, show_shapes=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-e5v5OOM1BHL",
        "colab_type": "text"
      },
      "source": [
        "### Check number of parameters\n",
        "\n",
        "We can also check the total number of trainable parameters of the model by calling `count_params()` on each result element of `model.trainable_weights`.\n",
        "\n",
        "According to **[iv]** there are 22,855,952 trainable parameters at Xception model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MvbYU04G1Bjr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import tensorflow.keras.backend as K\n",
        "np.sum([K.count_params(p) for p in model.trainable_weights])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AR7hVYiM1Etc",
        "colab_type": "text"
      },
      "source": [
        "## Final code\n",
        "\n",
        "```python\n",
        "from tensorflow.keras.layers import Input, Conv2D, SeparableConv2D, \\\n",
        "     Add, Dense, BatchNormalization, ReLU, MaxPool2D, GlobalAvgPool2D\n",
        " \n",
        "def conv_bn(x, filters, kernel_size, strides=1):\n",
        "    x = Conv2D(filters=filters,\n",
        "               kernel_size=kernel_size,\n",
        "               strides=strides,\n",
        "               padding='same',\n",
        "               use_bias=False)(x)\n",
        "    x = BatchNormalization()(x)\n",
        "    return x\n",
        " \n",
        " \n",
        "def sep_bn(x, filters, kernel_size, strides=1):\n",
        "    x = SeparableConv2D(filters=filters,\n",
        "                        kernel_size=kernel_size,\n",
        "                        strides=strides,\n",
        "                        padding='same',\n",
        "                        use_bias=False)(x)\n",
        "    x = BatchNormalization()(x)\n",
        "    return x\n",
        " \n",
        " \n",
        "def entry_flow(x):\n",
        "    x = conv_bn(x, filters=32, kernel_size=3, strides=2)\n",
        "    x = ReLU()(x)\n",
        "    x = conv_bn(x, filters=64, kernel_size=3)\n",
        "    tensor = ReLU()(x)\n",
        " \n",
        "    x = sep_bn(tensor, filters=128, kernel_size=3)\n",
        "    x = ReLU()(x)\n",
        "    x = sep_bn(x, filters=128, kernel_size=3)\n",
        "    x = MaxPool2D(pool_size=3, strides=2, padding='same')(x)\n",
        " \n",
        "    tensor = conv_bn(tensor, filters=128, kernel_size=1, strides=2)\n",
        " \n",
        "    x = Add()([tensor, x])\n",
        "    x = ReLU()(x)\n",
        "    x = sep_bn(x, filters=256, kernel_size=3)\n",
        "    x = ReLU()(x)\n",
        "    x = sep_bn(x, filters=256, kernel_size=3)\n",
        "    x = MaxPool2D(pool_size=3, strides=2, padding='same')(x)\n",
        " \n",
        "    tensor = conv_bn(tensor, filters=256, kernel_size=1, strides=2)\n",
        " \n",
        "    x = Add()([tensor, x])\n",
        "    x = ReLU()(x)\n",
        "    x = sep_bn(x, filters=728, kernel_size=3)\n",
        "    x = ReLU()(x)\n",
        "    x = sep_bn(x, filters=728, kernel_size=3)\n",
        "    x = MaxPool2D(pool_size=3, strides=2, padding='same')(x)\n",
        " \n",
        "    tensor = conv_bn(tensor, filters=728, kernel_size=1, strides=2)\n",
        "    x = Add()([tensor, x])\n",
        " \n",
        "    return x\n",
        " \n",
        " \n",
        "def middle_flow(tensor):\n",
        "    for _ in range(8):\n",
        "        x = ReLU()(tensor)\n",
        "        x = sep_bn(x, filters=728, kernel_size=3)\n",
        "        x = ReLU()(x)\n",
        "        x = sep_bn(x, filters=728, kernel_size=3)\n",
        "        x = ReLU()(x)\n",
        "        x = sep_bn(x, filters=728, kernel_size=3)\n",
        " \n",
        "        tensor = Add()([tensor, x])\n",
        " \n",
        "    return tensor\n",
        " \n",
        " \n",
        "def exit_flow(tensor):\n",
        "    x = ReLU()(tensor)\n",
        "    x = sep_bn(x, filters=728, kernel_size=3)\n",
        "    x = ReLU()(x)\n",
        "    x = sep_bn(x, filters=1024, kernel_size=3)\n",
        "    x = MaxPool2D(3, strides=2, padding='same')(x)\n",
        " \n",
        "    tensor = conv_bn(tensor, filters=1024, kernel_size=1, strides=2)\n",
        " \n",
        "    x = Add()([tensor, x])\n",
        "    x = sep_bn(x, filters=1536, kernel_size=3)\n",
        "    x = ReLU()(x)\n",
        "    x = sep_bn(x, filters=2048, kernel_size=3)\n",
        "    x = ReLU()(x)\n",
        "    x = GlobalAvgPool2D()(x)\n",
        "    x = Dense(units=1000, activation='softmax')(x)\n",
        " \n",
        "    return x\n",
        " \n",
        " \n",
        "input = Input(shape=[299, 299, 3])\n",
        " \n",
        "x = entry_flow(input)\n",
        "x = middle_flow(x)\n",
        "output = exit_flow(x)\n",
        " \n",
        "from tensorflow.keras import Model \n",
        "model = Model(input, output)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CUl25aXV0SB0",
        "colab_type": "text"
      },
      "source": [
        "## Model diagram\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/Machine-Learning-Tokyo/CNN-Architectures/master/Implementations/Xception/Xception_diagram.svg?sanitize=true\">"
      ]
    }
  ]
}